The field of language models has made significant progress in recent years. Large models like GPT-4 [17] have shown initial signs of general intelligence [8], while smaller models have demonstrated good reasoning abilities by solving challenging coding and math problems [11, 15, 16]. In this paper, we focus on the ability of small language models to solve grade-school math problems. Unlike previous works that empirically push the accuracy of models on grade-school math benchmarks like GSM8K [9] and its augmentations (e.g., [16, 22]), we take a more principled approach. We aim to understand the following fundamental questions: 1. How do language models learn to solve grade-school level math problems? Do they just memorize templates, or do they learn reasoning skills similar to humans? Or do they discover new skills to solve the problems?